feat(ai-observability): add time-bucketed breakdown to personal spend endpoint#69891
Merged
Merged
Conversation
Opt-in `hourly=true` query param adds a `by_hour` breakdown to GET /api/llm_analytics/@me/spend/: an hour-ascending UTC series with per-hour cost split into uncached input / output / cache read / cache creation components (plus matching token sums), capped to windows of 8 days or less. Enables clients to render a 24h view where prompt-cache cold starts stand out as cache-creation-dominated bars. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
Contributor
|
Hey @webjunkie! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Contributor
🤖 CI report✅ Bundle size — 🟢 -5.30 MiB (-7.6%)Uncompressed size of every built Total: 64.76 MiB · 🟢 -5.30 MiB (-7.6%)
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 280.3 KiB | ../node_modules/.pnpm/posthog-js@1.400.1/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 222.7 KiB | ../node_modules/.pnpm/posthog-js@1.400.1/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 92.7 KiB | ../packages/quill/packages/quill/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Dist folder size — 🟢 -128.97 MiB (-9.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1307.78 MiB · 🟢 -128.97 MiB (-9.0%)
ℹ️ ClickHouse migration SQL — 1 migration(s)
ClickHouse migration SQL per cloud environment
- unset
- all
DROP TABLE IF EXISTS property_values_mv
CREATE MATERIALIZED VIEW IF NOT EXISTS property_values_mv TO property_values AS SELECT team_id, property_type, property_key, property_value, property_count, coalesce(_timestamp, now()) as last_seen FROM posthog_test.kafka_property_values
- all
- US, EU
- aux
DROP TABLE IF EXISTS property_values_mv
CREATE MATERIALIZED VIEW IF NOT EXISTS property_values_mv TO property_values AS SELECT team_id, property_type, property_key, property_value, property_count, coalesce(_timestamp, now()) as last_seen FROM posthog_test.kafka_property_values
- aux
- DEV
- data
DROP TABLE IF EXISTS property_values_mv
CREATE MATERIALIZED VIEW IF NOT EXISTS property_values_mv TO property_values AS SELECT team_id, property_type, property_key, property_value, property_count, coalesce(_timestamp, now()) as last_seen FROM posthog_test.kafka_property_values
- data
2 tasks
…inutes Replace the hourly=true boolean with a bucket_minutes param (5, 15, 30, or 60) so clients can pick sub-hour resolution: 5-minute buckets match the prompt-cache TTL, isolating a cold-revival spike that hourly buckets would dilute. The by_hour breakdown becomes by_bucket (row field bucket_start, bucket size echoed) grouped via toStartOfInterval, and the window cap is now a flat 600 buckets of the chosen size. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
webjunkie
changed the title
…utes Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
webjunkie
marked this pull request as ready for review
July 10, 2026 10:29
Contributor
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
Contributor
|
Reviews (1): Last reviewed commit: "chore(mcp): update personal-spend tool s..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+168
to
+172
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, | ||
| allow_null=True, |
Contributor
There was a problem hiding this comment.
This field is optional, but marking it nullable makes generated clients expose bucket_minutes: null as a valid value. The frontend URL builder serializes that as bucket_minutes=null on a GET request, while DRF only accepts a real null value, so typed callers can send an allowed client value and receive a 400 instead of the bucketless response.
Contributor
Author
There was a problem hiding this comment.
Accepted in edbb2d9: dropped allow_null so the schema no longer advertises null (default=None stays for the omitted case), and the EU proxy now omits None-valued params from the signed body so the internal receiver accepts it.
Comment on lines
+797
to
+807
| from_dt, to_dt = _resolve_window(date_from, date_to) | ||
| if bucket_minutes is not None and (to_dt - from_dt).total_seconds() > bucket_minutes * 60 * MAX_TIME_BUCKETS: | ||
| max_hours = bucket_minutes * MAX_TIME_BUCKETS // 60 | ||
| raise exceptions.ValidationError( | ||
| { | ||
| "bucket_minutes": ( | ||
| f"A window this large would exceed {MAX_TIME_BUCKETS} buckets at {bucket_minutes}-minute " | ||
| f"resolution — narrow the window to {max_hours} hours or less, or pick a larger bucket size." | ||
| ) | ||
| } | ||
| ) |
Contributor
There was a problem hiding this comment.
A window exactly equal to bucket_minutes * 600 passes this check, but partial buckets at both edges can still produce 601 bucket starts. _fetch_by_bucket then returns those 601 rows as untruncated, so clients can receive more buckets than the documented maximum.
Contributor
Author
There was a problem hiding this comment.
Accepted in edbb2d9: validation now counts the bucket starts the window touches (unaligned edges add partial buckets) instead of comparing durations, and the fetch/truncate limits match, so by_bucket can never exceed 600 rows. Added a boundary test where a 600-bucket-duration window with a half-hour offset (601 starts) is rejected.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in bucket_minutes query param to the personal spend endpoint (GET /api/llm_analytics/@me/spend/) so clients can request a sub-day, UTC time-bucketed spend series (by_bucket) with per-component cost/token breakdowns, enabling higher-resolution usage visualizations (for example, spotting prompt-cache cold-start “revival” spikes).
Changes:
- Add
bucket_minutesrequest validation, cache-key partitioning, EU redirect/proxy forwarding, and a new HogQL aggregation forby_bucketin the personal spend API. - Add ClickHouse-backed tests covering bucket sizing, window caps, cost-component grouping/defaults, and cache-key behavior.
- Regenerate OpenAPI-derived schemas/types and MCP tool wiring to include the new query param and response shape.
Reviewed changes
Copilot reviewed 4 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/mcp/tests/unit/snapshots/tool-schemas/llma-personal-spend.json | Updates MCP tool JSON schema snapshot to include bucket_minutes. |
| services/mcp/src/tools/generated/ai_observability.ts | Forwards bucket_minutes into the generated MCP tool request query. |
| services/mcp/src/generated/ai_observability/api.ts | Updates generated zod query schema/docstring to include bucket_minutes. |
| services/mcp/src/api/generated.ts | Updates generated TS API types to include by_bucket and the bucket_minutes param enum. |
| products/ai_observability/frontend/generated/api.ts | Updates generated frontend API client docs/types to mention bucket_minutes / by_bucket. |
| products/ai_observability/frontend/generated/api.schemas.ts | Updates generated frontend schema types for the new by_bucket breakdown. |
| products/ai_observability/backend/api/test/test_personal_spend.py | Adds ClickHouse tests for by_bucket, window caps, and cache key behavior. |
| products/ai_observability/backend/api/personal_spend.py | Implements request param validation, new bucketed HogQL query, response serialization, caching, and EU forwarding. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
166
to
+180
| help_text="If true, bypass the result cache and re-run the underlying queries against ClickHouse.", | ||
| ) | ||
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, | ||
| allow_null=True, | ||
| help_text=( | ||
| "When set, additionally return a `by_bucket` breakdown: a time-ascending UTC cost series for " | ||
| "the scoped product at this bucket size in minutes, with per-bucket cost split into uncached " | ||
| "input / output / cache read / cache creation components plus the matching token sums. " | ||
| f"Supported bucket sizes: {', '.join(str(c) for c in BUCKET_MINUTES_CHOICES)}. The window may " | ||
| f"span at most {MAX_TIME_BUCKETS} buckets of the chosen size (e.g. 48 hours at 5-minute " | ||
| "buckets)." | ||
| ), |
Contributor
Author
There was a problem hiding this comment.
Fixed in edbb2d9: docs and the constants comment now say 50 hours at 5-minute buckets / 25 days hourly. The regenerated OpenAPI/MCP artifacts follow via CI codegen.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The test used the default 30-day window, which at 60-minute buckets is 720 buckets — rejected by the endpoint's own 600-bucket cap with a 400. Pin the window to a day so the test exercises the cache-key behavior it targets instead of the window validation. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
Drop allow_null from bucket_minutes: a nullable query param makes generated clients advertise null, which serializes to the literal string "null" in a GET query and gets rejected. The EU proxy now omits None-valued params from the signed cross-region body accordingly. Cap validation now counts the bucket starts the window touches (unaligned edges add partial buckets) instead of comparing durations, so by_bucket can never return more than 600 rows; fetch and truncate limits match. Also correct the cap arithmetic in docs (600 buckets is 50 hours at 5-minute resolution, not 48) and drop em dashes from the added API text. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
carlos-marchal-ph
approved these changes
Jul 14, 2026
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, |
Contributor
There was a problem hiding this comment.
I think it's right
| ) -> str: | ||
| to_slot = date_to or "_now" | ||
| return f"personal_spend:{email}:{date_from}:{to_slot}:{product}:{limit}" | ||
| return f"personal_spend:{email}:{date_from}:{to_slot}:{product}:{limit}:{bucket_minutes or 0}" |
Contributor
There was a problem hiding this comment.
Could we make it so we only append the :{bucket_minutes} part if the argument is present, so existing cache keys remain valid?
Contributor
Author
There was a problem hiding this comment.
Done in af97495: the :{bucket_minutes} slot is now appended only when the param is set, so bucketless requests keep their pre-existing cache keys and warm entries survive this deploy.
Drop the serializer default (None) so drf-spectacular emits no default for the param and generated clients cannot read it as nullable; omitted now means the key is absent from validated_data, with the compute function defaulting it. Append the cache-key bucket slot only when the param is set, so pre-existing bucketless cache keys remain valid. Also refresh the MCP tool-schema snapshot against the regenerated types. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…lback cost folding date_to is exclusive (timestamp < date_to), so a window whose end lands exactly on a bucket boundary can never produce a row in that final bucket; the cap check no longer counts it, letting users request the full advertised 600-bucket window. Added a boundary test. Also document on input_cost_usd that the uncached split only holds for gateway-provided cost breakdowns: events priced by PostHog's ingestion pipeline fold cache costs into input cost and leave the cache columns at 0. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The stored $ai_input_cost_usd includes prompt-cache read/write costs: on events carrying the cache cost columns, input + output reconciles to total and the cache costs sit inside input. Summing all four component columns therefore double counted cache spend, inflating stacked charts. by_bucket now derives uncached input per event as input minus cache read/write, clamped at zero so a future switch to exclusive reporting degrades to undercounting instead of double-subtracting. The four components are now disjoint and sum to cost_usd when the breakdown is present. The test fixture encodes the real inclusive semantics, so the assertions fail without the derivation. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
… by_bucket input_tokens claimed to be uncached, but whether cached tokens are included in $ai_input_tokens follows the provider's reporting ($ai_cache_reporting_exclusive): Anthropic-style events exclude them, OpenAI-style events include them. Describe the raw semantics and warn against stacking with the cache token sums instead of normalizing on a flag that is not reliably present. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2cbd7c434f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Same cache as the US compute path, so repeat loads skip the cross-region hop. | ||
| cache_key = _cache_key(email, data["date_from"], data["date_to"], data["product"], data["limit"]) | ||
| cache_key = _cache_key( | ||
| email, data["date_from"], data["date_to"], data["product"], data["limit"], data.get("bucket_minutes") |
There was a problem hiding this comment.
When the EU proxy serves a cached bucketed response, it returns before _compute_spend_analysis reruns the resolved-window cap check. With a relative end like date_to=-0hStart, the same raw cache key can be valid and cached just before an hour boundary, then resolve to more than 600 hourly buckets just after the boundary but still return the cached 200 for up to 5 minutes. Validate the resolved bucket count before this cache lookup, or key the cache by resolved bounds.
Useful? React with 👍 / 👎.
Contributor
Author
There was a problem hiding this comment.
Accepted in 232ebd3: the resolve-and-validate window check is now a shared helper the EU proxy runs before its cache lookup, so a relative window that resolves over the cap 400s immediately instead of serving a stale cached 200 across the boundary. Side benefit: EU callers now get fast in-region 400s for all invalid windows instead of paying the cross-region hop. Test added asserting the over-cap request is rejected without an upstream call.
…he cache lookup The EU proxy caches on the raw date strings, so a relative window cached while under the bucket cap could keep serving after it resolved over the cap. The resolve-and-validate step is now a shared helper the proxy runs before its cache lookup, which also gives EU callers fast in-region 400s for invalid windows instead of paying the cross-region hop. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
webjunkie
enabled auto-merge (squash)
July 15, 2026 09:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The personal spend endpoint (
GET /api/llm_analytics/@me/spend/) only exposes a daily total-cost series (by_day), so clients like the PostHog Code desktop app can't render a fine-grained usage view. In particular, prompt-cache cold starts — where a stale session is revived and its whole context is re-written to the cache at the cache-write rate instead of being read back cheaply — are invisible: they need sub-hour resolution and a per-component cost split to stand out. Since the prompt-cache TTL is 5 minutes, a 5-minute bucket is roughly one turn, which isolates a revival spike that hourly buckets would dilute.Why: we want the PostHog Code app to show a 24h usage chart granular enough that a sudden burst of full-price tokens (a revived cold session) is visually obvious. A companion PR in the PostHog Code repo (PostHog/code#3329) consumes this.
Changes
bucket_minutesquery param (5, 15, 30, or 60) on/api/llm_analytics/@me/spend/.by_bucket: a time-ascending UTC series (grouped viatoStartOfInterval) with per-bucketcost_usdplus four disjoint cost components (input_cost_usd,output_cost_usd,cache_read_cost_usd,cache_creation_cost_usd) and the matching token sums. The stored$ai_input_cost_usdincludes cache read/write costs, so uncached input is derived per event as input minus cache read/write (clamped at zero); the components sum tocost_usdwhen the breakdown is present, so they stack without double counting cache spend. The bucket size is echoed on the breakdown.by_bucketignoreslimitlikeby_daydoes.Note: CI codegen keeps the generated OpenAPI types current (no DB available for
hogli build:openapiin the authoring environment).How did you test this code?
Added tests to
test_personal_spend.py, each catching a distinct regression:test_by_bucket_groups_cost_components_per_utc_hour— wrong column order/mapping or grouping in the new HogQL query, and product-scope leakage.test_by_bucket_five_minute_buckets_split_within_the_hour— the bucket size not actually being applied (two calls 15 minutes apart must split at 5-minute resolution).test_by_bucket_defaults_components_to_zero_when_breakdown_missing— fallback-priced events (total cost only) must produce zero components, not errors.test_by_bucket_absent_unless_requested—by_bucketleaking into default responses.test_cache_key_includes_bucket_minutes— a cached bucketless payload being served to a bucketed request (would silently dropby_bucket).test_bucket_window_cap(parameterized) +test_unsupported_bucket_size_rejected— the 600-bucket cap or the size allowlist being dropped.Could not run ClickHouse-backed tests in this sandbox (no database); verified locally that both files compile, the new HogQL parses via
parse_select(toStartOfInterval/toIntervalMinuteare whitelisted), and the response serializer omitsby_bucketwhen absent. CI runs the full suite.Automatic notifications
Docs update
Endpoint is documented via its OpenAPI schema (
@extend_schemadescription and serializerhelp_textupdated in this PR).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/improving-drf-endpoints,/writing-tests.$ai_generationevent per LLM call with LiteLLM's per-side cost breakdown, and this endpoint is the sanctioned user-scoped surface over those events — so the view was added here rather than as client-side HogQL (the events live in an internal analytics team, not the user's project).hourly=trueboolean; generalized tobucket_minutesafter deciding hourly resolution dilutes single-turn revival spikes (cache TTL is 5 minutes, so 5-minute buckets ≈ one turn).Created with PostHog Code